The present invention concerns image processing methods and apparatus and, in particular, a method and apparatus for processing a sequence of images to recover images of unchanging portions of the scene that are partially obscured by moving objects in the scene.
There are many applications in which it is desirable to identify unchanging portions of a sequence of video images. These include data compression systems, surveillance systems and image compositing systems. The unchanging portions of a sequence of images represents redundant information in all of the images except for the first image in which they appear. Sequences of video images often include a considerable amount of redundant temporal information. For example, in a television news broadcast the background and even much of the news announcer remains substantially the same from frame to frame. In a video surveillance system, the image is substantially the same from frame-to-frame except for possible intruders.
This temporal redundancy may be used to advantage when processing a sequence of images. For example, in image data compression systems, if the static portions of a sequence of images can be identified, they may be transmitted separately from the changing portions of the image, thus reducing the total bandwidth needed to transmit the sequence of images. In a surveillance system, the unchanging portions of the image may be subtracted from newly received images to provide an output image that contains only the inconsistencies in the newly received images with respect to the preceding background images.
Depending on the application, however, it may be difficult to identify the unchanging or background portions of the sequence of images. This difficulty occurs when foreground objects frequently obscure portions of the background image. One example of such in application is a traffic monitoring system. An exemplary system of this type may produce an image such as the one shown in FIG. 1. The image includes three automobiles 110, 112, and 114 travelling along a road 116. In order to monitor the traffic flow within the field of view of the image 100, it is desirable for the traffic monitoring system to recover an image of the road 116 without the cars 110, 112, and 114. The image of the road 116 is not constant, it changes over the course of the day, for example, with changes in solar illumination. Although the traffic monitoring system may need to obtain an image of the road in the absence of the automobiles, this may be difficult to do, especially if the road is very heavily traveled, as the image of the road will be visible only between the times that the cars are visible.
One solution to this problem is to pre-store images of the road surface corresponding to different times of day and different weather conditions. This method, however, is subject to error in the selection of the appropriate background image and may require the storage of a relatively large number of images in order to cover all times of day in all seasons of the year.
The present invention is embodied in a method and apparatus which robustly estimates the brightness or other feature values of an image region in a sequence of video images even when the region is obscured by objects over large portions of the video sequence. In accordance with the present invention, a histogram is generated for each image region over a plurality of image frames in the sequence. The mode, or most frequently occurring value, of each region as indicated by the histogram is selected as representing the unchanging portion of the image. The mode values of all of the pixels in the regions are then assembled to form a background image.
According to one aspect of the invention, the histogram is generated using a recursive filter.
According to another aspect of the invention, the images in the video sequence are aligned before generating the histogram.
According to another aspect of the invention, each image frame is filtered prior to the histogramming operation to remove image artifacts caused by the camera.
According to yet another aspect of the invention, the region of the image is processed at regular intervals, skipping interstitial frames, in order to obtain the histogram.
According to yet another aspect of the invention, the region of the image is processed at random irregular intervals in order to obtain the histogram.
According to yet another aspect of the invention, the histogram is applied over a small number of frames in order to generate a noise reduced image.